Discussion of “ Least Angle Regression ” by Efron

I have enjoyed reading the work of each of these authors over the years, so it is a real pleasure to have this opportunity to contribute to the discussion of this collaboration. The geometry of LARS furnishes an elegant bridge between the Lasso and Stagewise regression, methods that I would not have suspected to be so related. Toward my own interests, LARS offers a rather different way to construct a regression model by gradually blending predictors rather than using a predictor all at once. I feel that the problem of “automatic feature generation” (proposing predictors to consider in a model) is a current challenge in building regression models that can compete with those from computer science, and LARS suggests a new approach to this task. In the examples of Efron, Hastie, Johnstone and Tibshirani (EHJT) (particularly that summarized in their Figure 5), LARS produces models with smaller predictive error than the old workhorse, stepwise regression. Furthermore, as an added bonus, the code supplied by the authors runs faster for me than the step routine for stepwise regression supplied with R, the generic version of S-PLUS that I use. My discussion focuses on the use of Cp to choose the number of predictors. The bootstrap simulations in EHJT show that LARS reaches higher levels of “proportion explained” than stepwise regression. Furthermore, the goodnessof-fit obtained by LARS remains high over a wide range of models, in sharp contrast to the narrow peak produced by stepwise selection. Because the cost of overfitting with LARS appears less severe than with stepwise, LARS would seem to have a clear advantage in this respect. Even if we do overfit, the fit of LARS degrades only slightly. The issue becomes learning how much LARS overfits, particularly in situations with many potential predictors (m as large as or larger than n). To investigate the model-selection aspects of LARS further, I compared LARS to stepwise regression using a “reversed” five-fold cross-validation. The cross-validation is reversed in the sense that I estimate the models on